Collect and Offline Process

Before Solid can answer a business question, it first has to learn how your data is actually structured and used. The offline process is the background pipeline that does this — reading only metadata (schema structure, query history, usage patterns) from your warehouses, lakes, and BI tools, never your underlying business data, and compressing millions of historical queries down to the small set that reflects real, repeated business logic. It runs on a schedule, never in the request path of a question a user asks.

What the Offline Process Produces

The offline process does not build semantic models. It builds the foundation a semantic model is built from:

A documented catalog — tables, columns, and schemas annotated with descriptions and quality signals
Relationships between tables inferred from real usage, not just what was formally declared
Extracted metric definitions
An enriched business glossary
A searchable index over all of the above

What comes out is both a catalog and a graph. The catalog lets you search for a specific table, query, column, or metric directly. The graph — built from the relationships above — lets you start at any one of those and expand outward to the dimensions and supporting facts around it.

Semantic models are built on top of this foundation, on demand, scoped to a specific business use case — see Semantic Model Lifecycle for that process.

The Pipeline

Connect (warehouse / lake / BI / docs)
        │
        ▼
Collect ─── schema, query history, usage, non-PII samples, BI content, docs
        │
        ▼
Normalize ─ raw extracts become a typed asset catalog
        │
        ▼
Analyze ──── queries parsed, resolved against the real schema, clustered
        │
        ▼
Infer ────── relationships inferred from declared keys + real join behavior
        │
        ▼
Rank ─────── naming, usage, and freshness decide what's worth surfacing
        │
        ▼
Enrich ───── descriptions, synonyms, and metrics generated for what survives
        │
        ▼
Index ────── published to a searchable, per-customer catalog

What Solid Collects

Source	What Solid Collects	Why It Matters
Warehouses & lakes (e.g., Snowflake, Databricks)	Schemas, tables, columns, declared keys, and non-PII column value samples	Seeds the catalog and grounds SQL generation in what actually exists
Query history	Historical SQL run against the warehouse — a rolling window, 90 days by default on first connection	The primary signal for how tables are actually joined, filtered, and used
Usage & freshness	Which tables and columns are queried, how often, by whom, and how recently the underlying data changed	Separates the tables that matter from the ones that don't
BI content (e.g., Tableau, Looker)	Dashboards, reports, and the calculated fields and metrics defined in them	An independent, already-validated source of business logic — often the fastest path to a correct metric definition
Organizational knowledge	Wikis, documentation, and any other context you make available	Grounds the glossary and enrichment in your own terminology

Column profiling happens in two passes — schema first, then, once real usage shows which columns actually matter, a second pass to sample those specifically, rather than profiling every column in the warehouse up front. See Data Warehouses, Databases, and BI Tools for the full, current list of supported connectors.

How Queries Become Understanding

A warehouse's query log is mostly noise — exploratory one-offs, broken drafts, admin housekeeping. Solid's analysis separates the small fraction that reflects real, repeated business logic from everything else: filter, break down, cluster, and keep only what's repeatable.

Rather than treat each query as a single unit, Solid looks at the pieces that build it — how tables are joined, how results are filtered, which groupings are used — and looks for repetition in those pieces across many queries. A calculation a hundred analysts independently wrote the same way is a strong signal; one person's different version is an outlier, not a pattern worth keeping. Who ran the query matters too: queries that fail to execute, or come from users whose queries rarely succeed, count for less.

For example: if ninety-nine queries compute a daily average by dividing by 365, and one divides by 360, that one is flagged as an outlier and dropped — not because Solid knows what "365" means, but because it doesn't match the pattern everyone else converged on.

The result is a dramatic compression: tens of millions of logged queries typically yield on the order of ten thousand representative ones — complex, deduplicated, and clustered. See Query Qualification Pipeline for the full funnel.

Inferring Relationships

Not every warehouse has its relationships formally declared — plenty of tables are joined constantly in practice with no foreign key ever defined to say so. Solid infers relationships from four kinds of evidence, in order of trust:

Declared primary keys
Declared foreign keys
Observed join behavior — when two columns are joined together often enough, across enough distinct queries, that's treated as evidence of a real relationship even without a formal declaration
Statistical evidence from the column profiles themselves — uniqueness, null rates, and naming patterns typical of a key column

Where these sources disagree, declared metadata wins over inferred behavior, and observed behavior wins over statistical inference alone. This is how Solid can propose a correct join path even in a warehouse where the DDL alone would leave it a guess.

Deciding What's Worth Surfacing

A typical warehouse has far more tables and columns than are useful for answering business questions — staging tables, deprecated columns, one-off exports. Before anything is enriched or indexed, every asset is scored on three signals — naming, usage, and freshness — combined with any preferences you've set, and that score decides whether it's surfaced at all. This is what keeps the catalog, and eventually the semantic models built from it, focused on what's actually worth an analyst's or an agent's attention, rather than a mirror of every table in the warehouse.

Keeping the Catalog Current

The first connection runs a full pass, looking back across a 90-day window of query history by default. After that, Solid checks for updates on a schedule — commonly weekly, though it's configurable per connection — reading only what's new or changed and running it through the same filtering and analysis. Assets that disappear from a source (a dropped table, a retired report) are marked removed rather than left stale.

This refresh also powers drift detection: when a run surfaces a change relevant to an existing semantic model — a renamed column, a metric drifted from its certified definition — Solid raises it as a recommendation for a modeler to review, never applying it automatically. See Data Drift Recommendations for how those are surfaced and resolved.

What Solid Reads — and What It Doesn't

Collection is metadata-first: schema structure, query text, and aggregate, non-PII column samples used to make categorical filters more accurate. Solid does not store your raw business data, and PII is detected and excluded at collection time — before it's written anywhere in Solid's system. Every connection uses a dedicated, least-privilege credential scoped to exactly what you've authorized.

See When Does Solid Execute SQL? and Security Architecture for the full breakdown of what Solid can access, and under what identity, at every stage.